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This study examined air traffic controller acceptability ratings based on the effects of differing 
horizontal miss distances (HMDs) for encounters between UAS and manned aircraft. In a 
simulation of the Dallas/Fort Worth (DEW) East-side airspace, the CAS-1 experiment at NASA 
Langley Research Center enlisted fourteen recently retired DEW air traffic controllers to rate well- 
clear volumes based on differing HMDs that ranged from 0.5 NM to 3.0 NM. The controllers were 
tasked with rating these HMDs from “too small” to “too excessive” on a defined, 1-5, scale and 
whether these distances caused any disruptions to the controller and/or to the surrounding traffic 
flow. Results of the study indicated a clear favoring towards a particular HMD range. Controller 
workload was also measured. Data from this experiment and subsequent experiments will play a 
crucial role in the FAA’s establishment of rules, regulations, and procedures to safely and 
efficiently integrate UAS into the NAS. 

Unmanned Aircraft Systems (UAS) are no longer technological systems of the unforeseeable distant future, 
but rather of the present and near future. They are systems that are evolving quickly and will soon become 
commonplace in the National Airspace System (NAS). According to the Federal Aviation Administration (FAA) 
Modernization and Reform Act of 2012 (2012), the United States Congress mandated the FAA to open the NAS to 
civil UAS “as soon as practicable, but not later than September 30, 2015.” However, opening the NAS to civil UAS 
is a challenging task, a task that encompasses multiple safety issues of which include detect and avoid (DAA) 
implementations, self- separation (SS) procedures, and collision avoidance (CA) technologies to remain well-clear of 
other aircraft. Routine access to the NAS will require UAS to have new equipage, standards, rules and regulations, 
and procedures, among others, in addition to a slew of supporting research efforts. As a result, the National 
Aeronautics and Space Administration (NASA) has established a multi-center “UAS in the NAS” project, in 
collaboration with the FAA and industry, to examine essential safety concerns regarding the integration of UAS in 
the NAS. Among NASA’s guiding research efforts is NASA Langley Research Center’s (LaRC) air traffic 
Controller Acceptability Study (CAS) human-in-the-loop (HITE) experiment series. The first CAS experiment 
(CAS-1) researched a subset of safety features to examine well-clear volumes by simulating differing horizontal 
miss distances (HMDs) at the Dallas/Fort Worth (DEW) East-side airspace. 

The concepts of remaining well-clear and DAA come from current standards under which pilots currently 
operate within the NAS. According to Title 14, Part 91, Section 91.111 (a), of the Code of Federal Regulations 
(14CFR 91.111 (a)), “no person may operate an aircraft so close to another aircraft as to create a collision hazard,” 
and 14CFR 91.113 (b), under right-of-way rules, states ''General. When weather conditions permit, regardless of 
whether an operation is conducted under instrument flight rules or visual flight rules, vigilance shall be maintained 
by each person operating an aircraft so as to see and avoid other aircraft. When a rule of this section gives another 
aircraft the right-of-way, the pilot shall give way to that aircraft and may not pass over, under, or ahead of it unless 
well clear.” In essence, these standards, among others, require pilots to follow right-of-way rules and remain well- 
clear, by seeing and avoiding, other aircraft. In an Air Traffic Services (ATS) environment, pilots are expected to 
comply with those requirements while also complying with Air Traffic Control (ATC) instructions and clearances, 
or to negotiate changes, as necessary, to those instructions and clearances. Pilots capable of seeing and avoiding 
other aircraft are mostly expected to maneuver and communicate in predictable ways; ways that preserve the safety, 
orderliness, and efficiency of the ATS environment. Inherently, UAS pilots will be expected to operate in a similar 
manner. As such, in October of 2009, the term sense and avoid (SAA), used interchangeably with DAA and 
comparable to manned aircraft see-and-avoid requirements, was defined as “the combination of UAS Self- 
Separation (SS) plus Collision Avoidance (CA) as a means of compliance with 14CFR Part 91, §91.111 and 
§91.113” and published by the FAA-sponsored SAA for UAS Workshop Final Report. The SAA for UAS 
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Workshop Final Report goes on to define SS and CA as a means to remain well-clear and as a means to avoid Near 
Mid-Air Collisions (NMACs), respectively. Under Section 6, 7-6-3 (b), of the Aeronautical Information Manual 
(AIM), the FAA defines NMACs as “an incident associated with the operation of an aircraft in which a possibility of 
collision occurs as a result of proximity of less than 500 feet to another aircraft...” Figure 1 shows the different 
volumes and boundaries associated with remaining well-clear. In order to remain well-clear, the Self-Separation 
Volume (SSV) size should be large enough to avoid corrective Resolution Advisories (RAs) for Traffic Collision 
Avoidance System (TCAS) -equipped intruders; safety concerns for controllers; and, undue concern for proximate 
see-and-avoid pilots. Determination of minimum and maximum operationally acceptable SSV sizes will inform the 
design space for required DAA surveillance accuracy. Current standard NAS operations are the building blocks for 
which future UAS NAS operations will advance. 
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Figure 1. NASA’s Separation Assurance/Sense- 
and- Avoid Interoperability (SSI) SSV represents a 
concept volume of remaining well-clear. Note. 
CAT, SSV and SST boundaries are notional and 
generally not cylindrical. 


Controller Acceptability Study-1 Objectives 

The primary focus of the CAS-1 experiment was 
on determining the effects of self-separation 
maneuvering tasks, as performed by pilots in a 
Ground Control Station (GCS) using simulated DAA- 
equipped UAS, on ATC workload and how the 
resulting maneuvers impacted ATC acceptability of 
the differing spacing parameters, also known as 
HMDs, which were implemented in the DAA 
algorithms. 

The aim of CAS-1 was to address, through data 
collection and analysis, the following research 
questions: A) Are DAA SS maneuvers too small/too 
late, resulting in issuance of traffic safety alerts or air 
traffic controller perceptions of unsafe conditions?; B) 
Are DAA SS maneuvers too large (excessive “well 
clear” distances), resulting in behavior the air traffic 
controller would not expect and/or disruptions to 
traffic flow?; and, C) Are there acceptable, in terms of 
ATC ratings, workload, and closest point of approach 
data, DAA miss distances that can be applied to the 
development of DAA algorithms? 


In order to address the above research questions, an appropriate experiment design was necessary to 
achieve the goal of the experiment’s primary focus and aim. 


Method 


Subjects 

To keep in line with designing an appropriate experiment, ATC subjects who had real-world experience 
controlling the East- side area of DFW were sought after, and, as such, fourteen recently retired DFW controllers 
were utilized for this experiment. ATC experience among subjects ranged between 25.5 years to 33 years with an 
average of approximately 30.4 years. Subjects also had an average of approximately 20.4 years of DFW experience 
in a Terminal Radar Approach Control Facility (TRACON). Additionally, of that DFW experience, an average of 
18.3 years’ worth of experience was in the East-side sector of the DFW TRACON (DIO) region. Furthermore, out of 
the fourteen subjects, none had experience with UAS operations, which allowed for a fresh perspective to 
controlling UAS traffic encounters, and four of the fourteen controllers were active instructors at the DFW training 
center. Also, in order to maintain and simulate a close to real-world DFW environment and workload, two pseudo- 
pilots controlled each UAS GCS and two additional pseudo-pilots controlled background traffic. ATC positions, 
other than that of the subject controller, were ‘controlled’ via personnel acting as other DFW TRACON sector 
controllers. The subject controller was expected to communicate with these other sectors as he normally would in 
the field, with the exception of some Standard Terminal Automation Replacement System (STARS) functions; 
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STARS “provides controllers with critical operational information about aircraft positions, flight data, and weather” 
(FAA, 2012). 

Independent Variables 

With the aim of acquiring data on ATC acceptability ratings on differing spacing parameters, the primary 
Independent Variable (IV) of interest was determining the minimum acceptable HMD as a result of a given 
parameter in the DAA algorithm. The secondary IV of interest was the encounter geometry between the aircraft in 
the encounter situation. 

Horizontal miss distances. CAS-1 researched six different HMD values that included the following 
spacing parameters measured in nautical miles (NM): 0.5, 1.0, 1.5, 2.0, 2.5, and 3.0. These values were implemented 
in the DAA algorithm. 

Encounter geometry. Three encounter geometries were utilized in CAS-1, which included opposite- 
direction, overtake, and crossing. Figure 2 visually portrays the different encounter geometries. The following 
parameters frame the secondary IV : 

• Intruder opposite-direction at 180 degrees +!- 15 degrees (non-crossing) 

• Intruder to right at 90 degrees +!- 15 degrees (crossing) 

• Intruder ahead at 0 degrees +!- 15 degrees (overtaking, non-crossing) 

• All geometries without vertical separation (but may include climbing/descending trajectories) 

• UAS pilots were instructed to pass to the right of intruder for non-crossing geometries 

• UAS pilots were instructed to pass in front of intruder for crossing geometries 

• Intruder Speed Differential (5 speed values for crossing: 0, 40, - 40, 80, and - 80 knots) 

• 42 test conditions: 6 opposite-direction, 6 overtake, 30 crossing 

• 14 encounters per hour; 6 one-hour test sessions per subject enabled a replicate for each encounter 

The parameters of the primary and secondary I Vs are shown in Table 1. 


Table 1. 

Parameters of the primary and secondary independent variables. 


Encounter Geometry 

Horizontal Miss Distances in Separation Algorithm 

0.5 

1.0 

1.5 

2.0 

2.5 

3.0 

Opposite-direction 

1 speed 

1 speed 

1 speed 

1 speed 

1 speed 

1 speed 

Overtake 

1 speed 

1 speed 

1 speed 

1 speed 

1 speed 

1 speed 

Crossing 

5 speeds 

5 speeds 

5 speeds 

5 speeds 

5 speeds 

5 speeds 






Scenarios 




The scenarios implemented in CAS-1 
simulated ATC Sector DN/AR-7 South Flow, 
which is a portion of airspace delegated to DFW 
TRACON (DIO). The scenarios were designed 
and situated in the selected airspace so as to 
enable various encounter geometries between the 
UA and intruder aircraft. 


Figure 2. Encounter geometries used in CAS-1 
included, from left to right, opposite-direction, 
overtake, and crossing encounters. 
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Dependent Variables 


System Performance Metrics. Aircraft-to-Aircraft separation distances, operational errors and deviations, 
delays to aircraft in scenario, re-sequencing arrival aircraft, and voice communication errors, which included 
transposing information, call sign errors, repeats, and “say again” were recorded during each one -hour data 
collection run. 

Human Operator Performance Metrics. Three different human operator performance metrics were 
examined. Among those three was the assessment of controller workload through the use of the Air Traffic 
Workload Input Technique (ATWIT) methodology. ATWIT was the tool used to measure mental workload in “real- 
time” by presenting auditory and visual cues that prompted the controller to press one of six ratings at fixed time 
intervals to indicate the amount of mental workload experienced at that moment (Stein, 1985). The response scale 
was built into the controller display software and had ratings from 1 to 6. A rating of 1 suggested ‘‘minimal mental 
effort required’^ a rating of 2 suggested “low mental effort required’^ a rating of 3 suggested “moderate mental 
effort required’^ a rating of 4 suggested “high mental effort required'^ a rating of 5 suggested “maximal mental 
effort required’^ and, a rating of 6 suggested “intense mental effort required'' In addition, another performance 
metric collected involved post encounter verbal queries that were gathered to evaluate controller acceptability of 
HMD spacing parameters. Controllers were asked to rate HMDs based on a scale from 1-5. Table 2 shows the 
scaled used and defines each of the acceptability ratings. Lastly, an “end-of-hour questionnaire” was administered to 
each subject controller at the conclusion of each one-hour data collection session. 

Facilities, Software, and Hardware 

The experiment was conducted in a dedicated facility located at Stinger Ghaffarian Technologies (SGT), 
near NASA LaRC in Hampton, Virginia. The facility ran a UAS modified version of the Multi Aircraft Simulation 
System (MACS) software (Prevot, 2002). MACS is an environment for developing, setting up, and running real- 
time controller and pilot-in-the-loop simulations; it was configured to emulate the existing Air Traffic Management 
(ATM) system. The modified version of MACS included incorporation of UAS aircraft models with the addition of 
Stratway-F algorithms to drive the Electronic Horizontal Situation Indicator (EHSI), known as bands, which 
indicated a range of headings that would result in a loss of well-clear with one or more intruder aircraft. Munoz, 
Narkawicz, Chamberlain, Consiglio, and Upchurch (2014) provide additional information regarding self- separation 
algorithms. The subject controller’s workstation closely resembled the workstations that are currently used in FAA 
field facilities. STARS functionality was included in this experiment but with limitations. The implementation team 
included personnel from SGT, Adaptive Aerospace Group (AAG), and Intelligent Automation Inc. (lAI). 

Table 2. Results 

Horizontal miss distance acceptability rating scale. 

Horizontal Miss Distances 


Subject controllers were verbally asked to 
rate HMDs on a scale from 1-5, as shown in 
Table 2, based on their acceptability of the 
HMD spacing parameter. 

Opposite-direction encounters. 

Illustrated in Eigure 3, the ratings for the 
opposite-direction encounter geometry show 
that HMDs with a spacing parameter of 3.0 
NM were considered unacceptable due to 
either being “somewhat wide" or “excessively 
wide." In addition, the graph also shows that 
the HMDs that the controllers’ found to be 
acceptable were the ones in the 1.0 and 1.5 
NM range with 80% of ratings suggesting 1.5 
NM being the most acceptable among the two. 



Horizontal Miss Distance Rating Scale Definition 

Rating Scale 


1 

Much too close; unsafe or potentially so; cause or 
potential cause for issuance of a traffic alert 

2 

Somewhat close; some cause for concern 

3 

Neither unsafely close nor disrupt! vely large; did not 
perceive the encounter to be an issue 

4 

Somewhat wide, a bit unexpected; might be 
disruptive or potentially disruptive in congested 
airspace and/or with high workload 

5 

Excessively wide, unexpected; disruptive or 
potentially disruptive in congested airspace and/or 
with high workload 
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Overtake encounters. Figure 4 illustrates the 
ratings for the overtake encounter geometry. The 
graph shows that the highest percentages, with a 
rating of 3, were at the 1.0, 1.5, and 2.0 HMD 
spacing parameters. In addition, the graph also 
shows that a rating of more-than 3 was given for 
HMDs with a 2.5 or 3.0 spacing parameter. 


>3 % 


Crossing encounters. Figure 5 illustrates the 
ratings for the crossing encounter geometry. The 
graph affirms that the controllers found the 1.0 and 
1.5 NM HMD spacing parameters to be the most 
acceptable by giving a large majority of encounters, 
with those specific spacing parameters, a rating of 
3 indicating that they were ‘‘neither unsafely close 
nor disruptively large'' and “did not perceive the 
encounter to be an issue." HMDs of 2.5 NM had 

comparable percentage ratings of 3 and more-than 3. Furthermore, as was the case with the other two encounter 
geometries, HMDs with 3.0 NM spacing parameters, received a majority of ratings of more-than 3, indicating that 
those encounters were either “somewhat wide," or “excessively wide" and “disruptive." 


Horizontal Miss Distance (NM) 

Figure 3. Subject controllers’ ratings for HMD spacing 
parameters for the opposite-direction encounter geometry. 
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Figure 4. Subject controllers’ ratings for HMD 
spacing parameters for the overtake encounter 
geometry. 


Figure 5. Subject controllers’ ratings for HMD 
spacing parameters for the crossing encounter 
geometry. In this crossing encounter, the UA’s 
speed was faster than the encounter aircraft. 


In summary, the analysis of the data collected concludes that 1.0 to 1.5 NM were the most favored HMDs. 
It also concludes that the majority of subject controllers found that 0.5 NM to be considered “much too close" for all 
three encounter types. Furthermore, a majority of controllers found that 2.0 NM was not unreasonable but that 2.5 
NM and above were considered disruptive. 


Realism of Traffic Density and Workload Ratings 


Careful consideration was taken in the design and realism of the simulation environment. Research was 
conducted to find the optimal traffic density allowable to achieve the aim of the study while maintaining as close to 
real-world densities as possible for a realistic simulation of the DFW East-side airspace. At the termination of each 
one-hour data collection run, an “end-of-hour questionnaire” was administered to each controller. Among the 
questions asked was one regarding the realism of the traffic density; controllers were asked to “rate the realism of 
the traffic density of the simulation during the preceding hour." The following responses are collective for all 
subjects for all six one-hour data collection runs: 0% of responses were that “Traffic Density was significantly 
higher than in real operations’," 1.2% of responses were that “Traffic Density was somewhat higher than real world 
operations’," 55.6% of responses were that “Traffic Density was about the same as would be found in real world 
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Ratings (Grouped) 




operations’,'' 42.9% of responses were that ''Traffic Density was somewhat lower than real world operations’," and 
0% of responses were that "Traffic Density was significantly lower than in real world operations." Table 3 shows 
the average workload ratings, captured at five-minute intervals using the ATWIT methodology, for all subjects and 
for all data collection runs. 

Table 3. 

Average Air Traffic Workload Input Technique (ATWIT) Workload Ratings. 


ATWIT Time Intervals {in seconds) 

Average 

Rating 

300 

600 

900 

1200 

1500 

1800 

2100 

2400 

2700 

3000 

3300 

1.37 

1.79 

1.84 

1.68 

1.93 

1.89 

2.15 

2.37 

2.08 

1.89 

2.01 


Discussion 

The CAS-1 research experiment employed a close-to-real world simulation of the DFW East- side airspace. 
The study focused on determining the effect of simulated DAA-equipped UAS on ATC workload, as well as, on the 
acceptability of maneuvers with differing HMD spacing parameters used in the DAA algorithms. The results of the 
study confirmed a clear favoring, from the ATC perspective, towards a particular HMD range, which was 1.0 and 
1.5 NM; this range was still favored even when maneuvers were required to maintain those horizontal miss distances 
and appeared to be the optimal range for ATC acceptability. In addition, controllers found the DAA integration 
concept as presented to be absolutely viable. ATC workload ratings using the ATWIT method showed that the 
controllers considered the simulated workload to require minimal to low mental effort given their experience with 
the DFW sector. 

Follow-on research studies in this series of experiments will focus on assessing the impact of modeled 
communication delays on the execution of SS procedures as defined in the CAS-1 experiment and the performance 
of the Stratway-F generated maneuver guidance in the presence of winds. In continuation of the aforementioned 
follow-on research, additional research studies will address minimum and maximum acceptable declaration times 
for projected well clear losses, from the perspectives of both the air traffic controller and the Unmanned Aircraft 
(UA) pilot. Data from the CAS-1 experiment and subsequent experiments are meant to play a crucial role in the 
FAA’s establishment of rules, regulations, and procedures to safely and efficiently integrate UAS into the NAS. 
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